A Black-box Approach for Response Quality Evaluation of Conversational Agent Systems
نویسندگان
چکیده
The evaluation of conversational agents or chatterbots question answering systems is a major research area that needs much attention. Before the rise of domain-oriented conversational agents based on natural language understanding and reasoning, evaluation is never a problem as information retrieval-based metrics are readily available for use. However, when chatterbots began to become more domain specific, evaluation becomes a real issue. This is especially true when understanding and reasoning is required to cater for a wider variety of questions and at the same time to achieve high quality responses. This paper discusses the inappropriateness of the existing measures for response quality evaluation and the call for new standard measures and related considerations are brought forward. As a short-term solution for evaluating response quality of conversational agents, and to demonstrate the challenges in evaluating systems of different nature, this research proposes a blackbox approach using observation, classification scheme and a scoring mechanism to assess and rank three example systems, AnswerBus, START and AINI. Keywords—Evaluation, conversational agents, Response Quality, chatterbots
منابع مشابه
Response Quality Evaluation in Heterogeneous Question Answering System: A Black-box Approach
The evaluation of the question answering system is a major research area that needs much attention. Before the rise of domain-oriented question answering systems based on natural language understanding and reasoning, evaluation is never a problem as information retrieval-based metrics are readily available for use. However, when question answering systems began to be more domains specific, eval...
متن کاملA neuro-data envelopment analysis approach for optimization of uncorrelated multiple response problems with smaller the better type controllable factors
In this paper, a new method is proposed to optimize a multi-response optimization problem based on the Taguchi method for the processes where controllable factors are the smaller-the-better (STB)-type variables and the analyzer desires to find an optimal solution with smaller amount of controllable factors. In such processes, the overall output quality of the product should be maximized while t...
متن کاملOptimization of Fermentation Time for Iranian Black Tea Production
The optimum fermentation times of black tea manufactured by two systems of Orthodox and CTC (cut, tear & curl) were investigated by measuring the quality parameters of black tea, like: theaflavin, thearubigin, highly polymerized substances and total liquid colour during the fermentation stage. Optimum fermentation times from the beginning of fermentation were determined to be 60 min and 15...
متن کاملBeyond the Black Box Approach to Ethics!; Comment on “Expanded HTA: Enhancing Fairness and Legitimacy”
In the editorial published in this journal, Daniels and colleagues argue that his and Sabin’s accountability for reasonableness (A4R) framework should be used to handle ethical issues in the health technology assessment (HTA)-process, especially concerning fairness. In contrast to this suggestion, it is argued that such an approach risks suffering from the irrrelevance or insufficiency they war...
متن کامل